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occurred using the Depattment of Education *sl records for th%'' 
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application and allocation processes, consideration is ^iven to the 
types of error measures, including the likelihood of occurrence; 
severity, op size> of -error; ^eed consequence; and allocation 
consequence. The following types of comparisons that were used for 
each data element 'are also discussed: conceptually eacact, cross-year, 
internal consistency, and cross-school. For each data element, a 
chart shows the arror measures and the type of data "sources u^d in 
the comparisons/ Quantitative and qualitative methods of causal 
analysis and da^a collection procedures and problems are also 
addressed, along with the guidelines for performing the compai^isons. 
Finally, the desults of the error measurement analysis are 
summarized. ijQcluded is the form u^d to record information at thi 
""institution , as well as sample FISAP forms for speci£ic institut: 
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/-Advanced. Technology is conducting a Titie^ IV, Quality Control Study for the 
Office of Student Financial Assistance (OSFA). One task in the sfUdy is an analysis of 
error in the irisfitutional process of applying for aid (the FISAP process). - This analysis 
will use data collected in two^irfiasess on^ at Eb, and Qne at the institutions. This is 
an interim report printing the results of the .data collection activities that took 
place 'using re<5^^rds available at the Department of Education (ED) prior to the 
institutional data collection. ,This pa'per uses the error measures, and analysis 
p/oce<iures described in. the, previous report, "AnalS^ses pf Error Associated with the 
Application and Allocation Aspects of the Campus-Based Programs," as a- base from 
which t* report on the analysis of the data collected. 

The report has six sections. SectiQn-2'is a description of the FISAP and . 
allocation process. Section 3^describes the types of error measures used and the types 
of comparisons used for each data element. Section 4 describes the data collection • 
procedures that, were* used and the pi?oblems that were associated with those 
procedures. . Section 5 recaps the kincSs of analyses that were performed and the. 
results and limitations ^of the analyses. The last section presents a summation of the 
results- ' ^ ■ " ' ' 



2.0* FISAP PROCESS AND POTENTIAL ERRORS 

• ' . ^ ^ 

- 

. The purposes of the FISAP are found in the two components of the form. The. 
Fiscal Operations tleport presents an accounting of funds in the three* Campus-Based 
programs. The institutions reftort how much money was received, how much was 
spent, how it was' spent, and any changes in the status of their accounts. The, 
Application part, the focus of this [analysis, cor*tains inftfS-mation needed to compute 
the bstitutioo's allocation of funds, ^e information includes: . 



\ 
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nurribtr^j of eligible financial aid applicants by income level, 
i:e/graduate statts, and dependency status 



The em:^lLnnent by undergraduate/graduate status.of traditionai. institutions 
and tn^ISuiBjae^of continuing ajid new students by month for those 

with- 



institutions witn 
'Annduht of tuition 



rtraditional calendars 

fee rev«iue by undergraduate/graduate status 
Amount of Pell funds expended- ' \ 
The amount of state and institutionally administered grant fund^ expended. 




2.1 OVERVIEW OF THE APi>UCATION PROCESS AND P9TENTIAL ERRORS 



The process is, on the i^ct, relatively simple. The-data submitted by institutions 
is edited at least twice by ED And is ched<ed by the institution. Hovsrever, in ^aiity 
tb^ process is complex ami ha^s several points in it >s^Kere errors can do occur. 
TIjese critical points in the process include: ' 



The compilation of information at the.institution 

The actual fUling out of the Application portion x>f the FISAP 

The data entry of the information at the contract of site 

Th^edit checks corducted on the data by ED 

The sending to the institution a report of the errors it has found 

The corrbcting/changing of information at the institution 

The editing and updating of the new dayi submitted by insjtitutions 

The computation of the tentative and final allocatidns. " 



In addition, at various points in the overall process, the FISAP can bes 



• Lost 

• Misplaced ' 

• Not completely filled out by the institution. 



r 

While these latter 5>oints are certainly causes of concern for eiisuring the smooth 
operation of the process, this^report is restricted to errors in the data elements on the 
application and the processing, and not on the receipt control and filing procedures at 
the institution or ED. . " 

There are many different ty'pes of errors that can be made. Some are easily' 
detected by computer edits such as arithmetic mistakes; others are nq$ so easily found 
because there, is only one source for that particular piece o^in|ormation. The types of 
errors that can occur include: , 

Connip.iling the wrong data for entry onto the FISAP 
Arithmetic mistakes ^ ' ^ 

Incorrect transcription of iMormation ohto the FISAP 
Incorrect data entry * * >^ 

Incomplete checking of data elecjjents at the institution ' ' 

Incomplete edits • ' ' 

Incorr&t allocation algorithm. ^ 

•J . • ■ ^ ■ " 



The applicatipn requires- data from many sources other than ttje financial aid 
office. Compiling the correct informatlonl pfc i requires coordination between two or 
more offices; This could lead to misuncfSHhding of -definitions, lack of adequate 
records, and so on. The figt^e the financial aid office receives may not^be the 
appropriate one, and the financial aid administrator would probably not check its 
validity. . . * , \ ' - . ^ ^ 



f The application is complex to fill out even by ^the most knowledgeable of aid 
administrators. Because t^ 'definitions on the application change periodically, and 
because there are internal instructions, there is a good~\lnance that transcription 
errors and arithmetic errors could occur. Most of these "should be detected either by 
the cqmputer edits at ED or through institutional review of^e error report' it receives 
from ED. ' ^ • ^ , ■ ' 
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Svery data preparation contractpis^' is required to meet a certain ievei of 
accuracy in its data efttry activity. Even nneeting the contractual obligation still 
means that errors in dafta entf y can occur. These errors should be c^ght either by the 
computer edits or by the institution. 



Perhaps the most difficult types of errors to find a^e tjjose that are embedded 
within the software of a conniputer.' In the FISAP process, ^e d^a that is submitted to 
ED is subject to about 150 computer edits that test mainly for range of valQes and 
Internal consistenoy. Also, the institutional'allocations are automatically "generated. 
Both of the processes are complex toVperformj the softw2u:e systems rn^ust be 
thbroughly tested. ^ . ' . , 

The next section goes into more detail, about the errors that can occur and the 
<fescription of the errdr measures that are used in the analysis. Not all types of errQrs 
are addressed in this interim report where the only information available at ED was 
reviewed and analyzed. The results of the'on-site dat^ collection, to be reported at a' 
ia'ter date, will ^xf^nd the dinal^sls and the'iss^s that can be addressed. 

2^ ALLOCATtON ^GORTTHM - 

The iform of e^ch of the institutional allocation funds is fl^sentialiy an algorithm 
of subtrdcting institutional resourcj^ from institutional costs. There are separate 
f5|rmuias for SEOG and for CW-S and NDSL because the former is a grant program and 
the latter are- se^lf -help progranjS. Tfti^ two formulas are: 

SEOG need = .75 x (Cavg. ttiitiOn and "fees for undergraduates) + (avg. 
, living cost;) x (thef number of eligible aid applicants) - 

^ , aggregate family contribution. - Pell expenditures - state 

* expenditures - .25 x institutional experKiitures ^ - > 

. «^ ^ 
CW-S and NDSL need , = ((avg. tuition- and | ys for undergraduates, 

• J . graduates) +• (avg. living cost of andergraduates, 

graduates) x (number of undergrafduate, gradu- 
ate aid applicants) - (aggregate family contri- 
N " butidn) 

The components of, the forjjiulas are all sell-explanatory except for the aggre- 
gate family contribution. For each, income category of eligible aid applicants, a • 

. . •. •• ^ • • BEST COPY ilLASLE 



family contribution amount is assigne^d. This annount differs across level of enrollment 
and depen^Jency status. To calculate the aggregate family contribution for an- 
institution, the number of eilgiblS aid applicants at each incoftie level is multiplied by 
th^ average family , contribution and* those products are then surfinned first across . 
income groups a^nd t^en across level of enrollment and depend«Ky status- * ' 

The result of ,the!b cal^b^ions is a need figure for SEOG and CW-S/NDSL. 
These amounts are then added acroSSvimtitutiqns Ittjthln the states to obtain a total 
' . state need for either SEOG or C'if-S/NDSlK^h^li^tiitiQn's nepd figure is divided by 
the total for the state resulting in the instltution'i^fercentage of total state need. The' 
allocation is based on this porportion as well as the proportion of an institution's need 
to the total need for all schoorls nationwide. ' 

' - . , ■ '•' >. ■ ■ • . 

3.0 ERROR MEASURES, COMPARISONS, A3>ib. ANAL YSIS PROCEDURES 

In the^revipus section wis describe, the data elements and algorithms to allocate 

Campus-Based funds to individual campuses. T^is section presents three descripPtlon of 

the types of analysis, me*asures, and comparisons to be performed for the application 
% • » » 

and allocation process. • 

IK . ■ ' ' ■ 
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3.1 TYPES OF ERROR MEASURES 



For any particular data element there are iour possible error measures. It will 
' not be possible to utilize all four Terror measures for each data element.^ These four 
error measures are: ^ , 



• 
• 



Likelihood of occurrer^re 

/ 

Severity, or size, or error 
Need consequence 
Allocation consequence- 



These four measures are briefly defined in the following paragraphs,^ and are used in 
. the summary figure to indicate the types of measures to be estimated for each da^ 
element. - i , 
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Ukelihoocl of occurrence is simply the proportion, or percentajge, of c^es 
(scjiools) for which a difference between t^e "application" value and the "comparison" 
v^li^e occurs and is greater than some tolerance. Estimation of this type oi error 
measure is feasible for all data elements and comparison''values. 

Severity or size of difference in error provides an estimate of the seriousness of 
differences between application values and- comparison values. These measures would 
be reported asnhe average er/or or average difference between the two values. As 
with the. occurrence likelihood measure, severity can be estimated for all data 
elements and comparison values- 

^ Need conse<iuence measured, estin^ated effects of data element errors \>n the 
measures of absolutes aggregate need calculated for each ^campus. This type of 
measure will oaiy be used for data elements for which the comparison values represent 
exact values. For example^ it is possible to estimate the need consequence of 
differences between reported enrollment apd HEGIS entpliment; however, the compa- 
rispn between the institutionally reported income ^distributions and average income 
distributions would not support estimation Qf need Consequence. • 

AUocatioh consequerkx is the next step after need ^sequence and would/^hly 
include data 'elemeftts for which exact comparisons ^ supported. Allocktion 
consequences may not follow tl\je« same, pattern as need consequences because of 
various hold-harmless and maintenance-of-effort considerations, and because*of the 
pattern of error across other schpols. " ; 

3.2 TYPES OF pDMPARBONS v , 

- There are four tyi>es.of comparison values which will be use* in this analysis: 

V , ■ ^ ■ 

• Conceptually exact . . ' ^ • . ^ 

• Cross-Ye^ ^ ' • ^ 
a Internal Consistency . 

•. Cross-School. -'f^ ' ^ . ' 



Brief d'eacj^iptions of fhese types of comparisons afe presented in the following 
paragraphs. In Section 3.3 these comparison types and the measure types ar-e brought 
together in a figure which summarizes tlje plans -for analysis of errors associated with 
the application arid allocation process^ v 

Conceptually exact comparisons involve data elements where an alternative data 
sdurce contains condeptualiy-comparabie ^^alues for the HSAP data element. Exam- 
ples would be undergraduate enroUmenf and Pell expeoditure& For d$t|^lements 
which have conceputally exact v^luesMt is. possible to utilize all four types of error ^ 
measures. • • , ■ - ■ ' 

,-• ■ • ; •• ■> . . , ; • 

Cfoss-Year comparisons involve valu^ from the October 1981 .FISAP- These 
comparisons ar-e not exact 4xut would be expected to identify potential error when 
cross-year cjijanges exceed reasonable tolerances- We would sissess the likelihood and 
severity of cross-year differences but would not estimate need consequence^ or 
"allocation consequences.^ ^ " V 

. IiA^iial Qxtsistency comparisons would involve r^latifinships between *<iata ' 
elements in the application section of the FlSAP. Typically, these coxnparisons would 
involve Reported toUls and calculated sums of^components. Likelihood and severity 
would pe reported for all internal consistency comparisons. \Whether* need and 
allocation consequences are'estimabie needs to be decided on a case4)y-case bsisis. 

• L 

Cross-school comparisons focus of whether or not the value reported by a school 
is within a reason^le range of what-was reported by similar schools. It is an inexact 
comparison but the distributions of these differences may highlight data elements 
needing corrective actions. Only severity of .occurrence error measures would be 
appropriate given the inexact nature of these typ>cs of comparisons. . * 

3.3 DATA ELEMENTS, ERROR MEA5I5RES, AND COMPARISONS 

I 

Figure 1 -summarizes the planned errqr measurement analysis. It» Contains a row 
for each application data item In the allocation process. The entries in each row 
indi'^ate either ikt type of error measure or the type o^ comparison to be performed 



. for each data element. 
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( FiGURE 1 
SUMMARY OF FISAP ANALYSES 
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The figure highlights the diversity of analyses, the da.t5i sources used, ana their 
..relationship. In some cases, due to limitations of directly comparable .dat^, only 
occurrence and severity ^measures of error can be constructed. In others,, the wiysis 
can pinpoint the eyentuai effect on an institution's allocation caused by an errpr in a 
t'data item. , • ' 



• 



3A CAUDAL. ANALYSIS 



4^ There §fje^two_ types of causal anaijcsis planned in order to assess potentialcaCises 
and to identify possible corrective ac;tion#. The^-* first type is the traditional, 
quantitative ^ssessment,^ while thtf^ond involves a more qualitative apfs^bash, 

3.4.r Quanti^tive Methdtfs of Caulh Ansa ' ' . . . . ' 

I '■ ' " • . . ■ - . * 

OTtis/type of analysis , primarily cwisists^of bivariate cross-.talsulations of data- 
element error-measures by institutional characteristics. • The nature of the reasjsarch 
quesfion^ 'Will beVwhether cr not the likeiiho<jd of 'occurrerke, severity oV size*of . 
differwiices, and need or allocation" cortsequeiy:es. dif fer across -the characteristics of 
th^s^oo^s. Existence of ^signtficarit differe^ees woulk suggest .'that thp associated 
instj^utionai characteristics (or correlates,4J^hat charaoteristic) caused the erforf 

/ Institutional characteristics to be investigated as causes or correlates of error " > 
»/w9uld- include: ' ^ . ' . ' . 



Type of institution • . . " ' - . * 

Institutional control ' ' 



Institutional office responsible fcr^ISAP preparation 



. • Method of preparation used 



Size of "institution 



• • ' ^Use of QC prbcedures ' 

if 

• . Characteristics of financial aid office 



• Jjjf Size of Cajnpus-B^d allocation ^ j 

. . • Federal program participation. . v nonv r-'Mr «ni r- 
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iA^2 Qualitative Methods of Causal Analysis . ' ^ s ' . 

*■"".• ' . * ■ 

Our proposed approach involves soliciting information directly frorrji the institu-* 
tion as to the causes of any errors we uncover^ using the comparisons arid measures 
discussed^ earlier. In order to optimally utilize scarce resources,' attention will be 
focused on the "exact" comparisons; Fbr any difference discovered during our pre- 
visit dat^ collec^on, we ,wUi ask the FAA to provide an explanation, rationalizajion, 
etc. Thus at any school this set of questions will be tailored to -the list of errors 
discovered for that school. ' f . ' / 



These i?|?^n-ended responses will be examined by project analysts in irder to 



/distill possibl^ iiauses of the errors and to suggaiC corrective actions. 
^.0 DATA COLLECTION PROCEDURES ' ^ ' 

The data collection actiyltie^ for anajyzing the FISAP are planned in two s^&s. 
The first step utilize.s the inf oi^mation that is available at the Department in the 
of program data or other files. The s€fcond step is to ask the institutions directly for 
the information they supplied on the FISAP. and^how they arrived at it- Using this 
appfoach, the first step of. data collection could be done before the institutionafffte 
visits. Any apparent errors discovered in the first stej^of data collection coyld then 
be explained by the visit to the institution.. This int'erim report is- confined to the 
results of the first data collection. * . . 

V ^ ; . - 

*.l DATA COLLECTION PLAN ' * • . . 

- The data colleciion activities were restricted to those sources that were readily 
available at ED/ Figure 1 from the previous section shows the data elements under 
analyst and the types of analysis to be used. -Data from the following sources were 
collected: . ^ * ■ . 

(T^ ' ... 

• Pell universe file 

'* , , ■ ' 

• fiigher Education General Information Survey (HEG^S) Fall Enrollment' 
■ survey ' ■* , 

I 

• HEGIS Financial Statistics survey 

■ • . . V"""^ CoPY f A'AILABLE 
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October, 19$! FISAP 



' • Postsecondary Schc5ols with Occupational Programs survey 

• ' Audits and pro'gr^m reviews , ' ' 

" . ' .. ■ » - 

• Hardcopy of the October,. 1982 FISAP. * 

The data collection plan was *to gather the value^f the relevant data items 
fr6m each data source. This was done, depending upon the kind of data, ifv the 
following ways: * * * 



• Abstracting from, general institutioMd files such* as audh;^and program 
reviews • . ' ■* " . " " * 

•.."./ , ■ ' • ■' . ■■ V ■ . ■, 

• Ejftracting data from program files such as the Pell Universe and the twQ 
FISAPs ' , - . 

• Extracting data from institutional surveys siK:h as the HEGIS and. the 
PostsecOTjdacy Schools with Occupational Programs surveys. 

Once ail the data had been gathered,, the values were entered onto summary data 
sheets, as shown in Figure 2. 

^ The summary data sheets contain all the' information necessary to perform the 
« ' • ■ ' > '■ ' . " 

initial stage of analysis. The data items of interest are listed down the left-hand side 

and the different sources of information listed across the top. Each occurrence of a 

data item for which there is a comparable yalue was then ej>tered in the appropriate 

box*. This was done for each of the 275 institutions in the sample. 

4.2 IN-HOUSE DATA COLLECTION'^' 

The actual data collection procedures are complex because of the multiple data^^ 
sources being utilized and because of the varied methods of analysis. Methods of data 
colie'fion varied according. to the form in which the data are found. For instance, the 
Pell Universe is on a computerized file and updated periodically so that form was the 
most efficient to use. The forms of the <lata we used are: 

* ■ ' ■ H- * 

/ ■ , . • 

. , ♦ \ ■ 

• Pell Universe file - cpmputerized data file with one record for each 
\ . instit^tlbn 



HEGIS Fall Enrollment survey -hard copy" of the HEGIS form for^ each 
institutipn 

. HEdlS Financial Statistics sXtrvey - h'ar^ copy of the HEGIS form 

October 19S1 FISAP - computerized data file with one record for each 
institution . ♦ 

' Postsecondary Schools with Occupat^oAa^Pro5ram$ survey - iSook contain- 
ing data on the schools in the survey ' « 

Audits - partially in folders and partially on microfiche with one folder or 

piece of microfiche for all the audits for th^t institution 

« • - 

Program reviews - microfiche with all program reviews on the microfiche 



r * 



* October 1982 FISAP - both the computerized data file and the hard copies 
from the ED files. , ' - # • ' 

The data collection went according to the plan in m^st respects. In most cases, 
the paper and microfiche files were in fairly good order ancl easily accessi)3le. The 
computerized files were very easy to^work with. The major problem with the paper 
files are their bulkiness and the fact they contain far more than one year of data. 
Moreover, tj>e paper files ar? difficult to keep track of in spite of attempts to 
maintain a. record of when files are taken out. The purposes of the files differ across 
the, various data files. As a result, the level of knowledge of the files ami the 

accessibility varies. , ^ ' 

■ _ ' • i . . -f : 

The two major problems with this data coiiection were the gathering of the most 
recent data, and conf-ectiy identifying the 'school we wanted. The first problem is 
endemic in any program because information is constantly updated. The second 
problem has been a perennial one in OSF A. 

m 

/, 

All of the data files we examined with the possible exception of the' Pell 
Universe begins with a hard copy form th^t is filled out,'key entered, edited, and then 
updated at least once. Since our^data collection examined both hard copy an4 
computerized files, it would be difficult to determine where any errors exist. For 
example, the HEGIS Fall Enrollment survey is keyed, edited, arKi then changed based 
on additional data collectidn. A revision made by the school may or. may not be. found 
in the hard copy fiiej it may be directly keyed in. Unfortunately, using the 
computerized file. may not be the most accurate because "revisions may be made to the 

, ' . . ■ ■ BEST COPY AVAILASLE 
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hard copy, but not keyed. 'All oi .the data files used in this data collection are subject 
to"" these potential errors. ' . 

The problem of identifying -the correct school is a complex one in this data 
collection. The profalenis of OSFA are compounded by the f that our sample of 
instittitions is not as well defined ^ we n'feeded. Thejdentlfication problem has two » 
components: f» common identifier across data files? and inconsistent reporting units 
across data files-^ The two types of problems overlap because vi{hile there may be a 
common identifier between t,wo data files, there is no guaraSntiee that multi-campus , 
schools report in the same grouping across files. The Pell and Campus-Based progranw 
have different identification, nyml^ers for the schools that participate in thj?. respective 
programs. Our data collection,Used the entity number (EIN) as a common identifier. 
For the other data fileaj we had to rely on the name of the school as the primary ^ 
identifier. This caused problems when schoo\s changed Rarp^ or when diff e&nt names 
are used for the different programs. This n:tay have con^ibuted to some of - the 
problems discussed in the next section. In much the same way» .fchools report 
differently^ to different offices of ED. The most common occurrence of this problem' 
is with branch campuses and omsortla of schools. There are no rules for defining a 
reporting unit so any attempt to identif 3^;^ particulaiflschool's information from different 
data sources may be im^ssibie. This, too, contributed to the missing data problems. 

*.3 PROBLEMS 

A number of problems were encountered in this p^iase of the data coileitlon. 
The two major types of problems include: " 

t Missing data elements, either the documents could not be found or the data 
eiementvyWas missing from the file 

• The data values found in the different sources did not compare as closely 
* to the definition used in the application and allocation process as was 
anticipated.^. 

Missing data occurred in two forms. In the first form, data elements were either 
not aggregated in the same way in the alternative source as on the FISAP, or the 
particular data value was missing. This type of missing data occurred with the Pell 
Universe and HEGIS files because schools use dif ferment reporting units for different 

% 
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programs. The second type of ^missing data was where there were no docunjents or 
trace of data. This happened primariiy when we were examining the hard copies of 
"documents for the data values. This type of missing data was found in *the hard copy 
FISAP, HEGIS, audits, and program reviews. 

The second major type of problem encountered in the data collectipm was where 
the definitions of the "data elements iivthe alternative sources do not match those used 
on the IflSAP. Theehect of this is to dilute thS power of the comparison of the two 
values, and hence the effectiveness of the anaiysiijr Some of the lack of directly 
comparable definitions were anticipated. TTie discission in* Section t identifes ^he 
data eiemeifts, such as tuition and fees, that Havejtomparabie, bu>= not exactlj^he 
same, definition. There were three unari^cipated definitional problems. 

. ^ ■ * ■ ' '• • 

One unanticipated problem with definitions occurred with the enrollment figures 
for nontraditional schools. The "^proposed alternative data source finally agreed upon 
was the Postsecondary Schools with. Occupational t»rograms survey. Other data 
sources were investigated, in particular the Vocational Education Data System (VEDS). 
How|?yer, it wai the^opinion of the NOES staff responsible for the data that VEDS does 
not collect the kind of data that is c«\tained'in the FISAP. Moreover, it was felt that 
a number of schools in our sample would TK>t b? found in VEDS. ^ ' . »^ 



The Postsecondary Schools with Occupational Programs survey i$ a biennial 
survey of schools that asks for basic identifying information about the schools, much in 
the same way as the Institutional Characteristics fUe of HEGIS. The survey coUects 
data on enrollment, cost of attendance, and other types of similar infprmation. The 
problems with using this survey for the purpose of the 1982 FISAP analysis are firsts 
that the relevant survey is for 19S0-S1, and second, theheadcount enrollment figure 
from the survey cannot be directly compared because the FISAP asks for*a breakdown 
of the number of new starts and continuing students by month. Initial attempts to • 
compare the headcount enrollment from the survey and a constructed *laverage" - 
enrollment from the FISAP proved to be unreliable. It is unknown whether the failure 
to adequately compare the values was due to the incomparabiiity of the data 
definitio^js, the one-year difference "between the data sources, or the inaccuracy of 
either the FI3AP or the survey data. 
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The secoryd unanticipated problem was the cross-y^ar comparison of institutional 
^ expenditures. The plan was to compare level of institutional expenditures repeated 
b'etween the October 1981 and OctoHjer 19S2 FISAP. It was thought that the llvelbf • 
^ . expenditures for the 1977-7S year would be reported -on the October i9|l fISAP as it * 

is on the October 1982 FISAP. This, in effect, would be a direct comparison since the ^ 
amouat for the i977-7S year should not change, between the two years. However, the 
October 1981 did not require the amount of institutional expenditures for the 1977-78 
year, but rather the amount for the 1980-81 academic y^; / 

The last unanticipated pf6blem • concerns the cross-year comparison of the 
maintenance of effort. The October 1981 FISAP asks for the level of institutional 
expenditures for 1981-82 for institutions that participated in CW-S or SEOG. The 
October 1981 FISA|* required all institut^hs to submit three years osf institutional 
expenditures. This means that the only valid comparison is between the 1981-82 
expenditure figure and th*. last year*^of'the three-year average, the 1^80-81 expendi- 
tures. ^ ^' 
* - ■ . ' - t ' 

The total effect^ of the problems encountered in the initial stage of data / 
collection is to restrict the scope of the analysis. By far, the major problem was the' 
missing document and/or missing data. For example, for 25 out of the 275 schools 
participating in the Campus-Based programs, no hard-<opy FISAPs could b^ found. 
- ' There could be many reasons for this including ED staff review of the FISAP without 
placing an "OUT" card- in the folder, misfiled FISAPs, and so forth. Extraordinary 
efforts to track down these 25 were not employed in order to minimize staff disruption 
and because of the shortage of time to complete the data coik^tion. 

DATA COLLECTION AT TJiE INSTmrnON • 



ERIC 



Once the pre-visif data collection ^ analysis was completed, a comprehensive 
profile of the application data in the FISAP was constructed. This provides the 
starting point for the data collection and analysis using data gathered at the 
institution. 

Foe each occurrence of an apparent error in the FISAP, a form was prepared for 
the institutional data collection. Figure shows the form that was used to record 
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S^PLEMENTAL FISAP QUESTIONS FpR 



As part of the Campus-Based Quality ^orlfp^i Study, Advanced Technology was asked 
by pe Department of Education to compare the values on the application portion, of 
the FISAP "tp alternative sdurces of information. We found what appear to be 
discrepancies in the following data items. Couid you explain the reason' for the 
apparent discrepancy and provide us.wlth any supporting documentation. 

Data Item pfeAP t Other Source 

Tuition and fees . ,» 7 '■ 

Expenditure - . 

IncorAlrl^ . • C ^ • 

' - . UG Dependent . , , 

^ Grad. Dependent ^ 
- UG Independent • 
Grad. Independ^t 

c 

UG Enrollment 
Grad. Enrollment 

' ■ ' " m 

\ 

Explanation tf 1 ( . ) - 




4i ' . • 

•Explanation tf2 C ) - 



Explanation #3 ( ) - 



FIGURE 3 



INSTITUTIONAL DATA COLLECTION SHEET 
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information at the institution. - Whenever there was an exception identified in the 
analysis, the FISAP value and the value from the aitemative .sotfrce w^re entered on 
the sheet along with the name of the alt'emative source. The data collectors were 
instructed during the^r training ^o ask tfie Institution's ^financial "^aid ^administrator 
(FAA) about the apparent discrepancy, to explain how th^ discrepancy could have 
occurred, and what procedures the institution uses to generate the values for the • 
FISAP^ , . ' ^ 

The gathering of informatibn*about4he diiscrepancies in the FISAP is an infegral » 

'< ■ 

^part of the overall data collection effort at thfe institution. The information can be 

. , ■ 

obtained using a combination of two activities thetiata collators will perform: 

■ ■ / ■ ■ ■ ■- ■• * 

• Through the questions' ^$i»it the FISAP in the Institution Questionnaire 

■ ■ ■ ■ . .V ' 

' Through requests for documentation for those data elements in the • 

application. .• 

- ■ \ ' ■ » 

* ■.. . 

^oth of these collection procedures will yield information about the values of the. 
items tfie institutions- us'fe in the application as well as the prccedures and calculations C 
Ms<id to prodiKie the informatiwi that goes into the\application. Examples of some 
returned institutional questionnaires appear in Att^ment As 

The results c|f the institutional data collection will then be tied back to the pre- 
visit collection and analysis to give an in-depth lopk at the causes of errors the 
. application and what the magnitude of the impact is of those errors. 

5.0 RJESULTS OF ANALYSIS . ' . 

* ■' ' ' ' ■ 

This section reports the results of examining the data collected at ED. Since the 

analyses are comprised entirely of comparisons, there is no specific plan of analysis 

per se. Instead, in the first part of the section there wUi be-a discussion of the general 

guidelines that were foilow«l while performing the comparisons. The second part of 

this section wii^ give an overview of the results of the analysis. ^ 

4r 
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5.1 METHOD OF ANALYSIS \ 

' Once'the data had been collected and entered onto the summary data sheet, the 
' cofT^pa'risons" of values began. Section 3 -identified the exact comparisons of data 
elements that can be madci and which comparisons are only -approximate. The exact 
comparisons are self-explianatory from an auiaiysis point' of view. In the approximate 
comparisons, the method^^4«ed becomes' important.^ 

■' ^ ^ ~ ' ■ \ t •. ' " 

A qumber of rules of thumb were used in ?h4f comparisons: • , ' 

^*-P8!" each data element for which there is an kpproximate comparison 
across two data sources^ for the same year, no error was reported if the 
values wferewilhirllO . percent of each other 

• For each data element for which there is a cross-year comparison, no error 
. . ■ was reported if the values were within 10 percent of each other 

• An error was reported if there was a difference between the value on the 
FISAP data file ami the hard copy FISAP 

• If either the enrollment or tuition and fefe revenues increased and the other 
' . decriJased by a signficant amount, and error was reported 

• • If the enrollment' increased and the^otal number of eligible aid applicants 
decreased "significantly, an error was reported. " , 

These rules were used aw guidelines in doing the comparisons- In some cases, whether 
or ndt to' report an error was purely a judgment call. The cross-year comparisons are 
particularly difficult because they are one-time snapshots of a difference. No trend 
analysis was developed that would make the evaluation of a ch^ge in value easier. 



5.2 RESULTS 



* 



The in-house data collection uncovered numerous error on the application portion 
of the FISAP foe a significant nunlber pf institutions in the sample. In total, SS errors 
at 83 institutions were found. Figure ^ shows the frequency of the errors by type of 
error. Enrollment errors dominate the kind of errors made. Over two-thirds of all 
errors are enrollment errors. The ne)tt highest is tuition and fejis error with twelve 
occurrences and then eligible applicant incomes with eight. The number of errors 
exceeds the number of school? because about 7 percent of the Institutions made more 
than one error. The types of errors are self -explanatory except for the tuition ^ 

' -19- 
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• fees/enroliment error. This error ocoirred six times- Thit err^r is a combination of 
"the change in tution and fees and.enrollment. If, for) instance, tuition and fee rbverye- 
. increases subst;antiaily whiie^nroiiment .drops, the school 'was flagged as ^eing in 
^rror. It would be impossible to det^mine which' one^s in error so the combination 
Vas defined as, an error. ^ . ^ ' ^ 

The schools thai made errors on the application come from ail sectors of 
postsecon^ary education. Figures, 5 shows the distribution of the 83 schools by type and 
control. Also presentd in Figure 5 is the distribution of schools who participated in the 
Camptis-Based programs in i9S3-84. As'can be seen, \t is rwt the proprietary schools 
who disproportionately N^make errors. Instead,' the pubgc institutions tend to make 
more than their share of errorsu This is-^potwitially disturbing .given the fact that 
public scho^>ls>receive such a large proportion of the funds. Attachment B shows the' 

* ♦ 

list of institutions that committed errors. ' , " - 

One intepesting item of note from the frequency of -errors is the relatively few 
number of institutions that committed more than one error* As^ mentioned above, only 
about 7 percent of the" institutions had two or oiore errors. This tends to run counter 
to our hypothesis that there are some schools that make errors all the way through the 
application* Instead, mbst schools made one error. Figure 6 shows th)f six schocfis that 
made more thauTone error. This list excludes those institutions that committed the 
tuition and fees/enrollment error and only includes those with unrelated errors. Figure 
6 shows that, like all the instif^tins making errors, those with multiple errors come 
from different sectors of postsecondary education. 

The high frequency of enrollment, er rocs could be due to one -or both of the 
following reasons: ^ 

•" The (sSmparison was exact and there was no room for discretion over small 
differences * t 

• The enrollment figures typically do not com4f|:om the financial aid office, ' 
but from the registrar or other administrative office, thus increasing the 
chance of error. 

The Pell expenditures vaiyes are also virtually exact comparisons, ^ut. usually the 
■ amount of Pell expervditures is maintained by the financial aid offlcdHHere is less 
chance for error. Moreover, the Pell amount can be taken directly from the Progress ^ 
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SCHOOL 



§t. Joseph's College (Private' ^Y^) 



..Rutgers (PubUc-Uriive^ily) 



ERRORS 



Tuition "and fees, enrollment 



Pell; enrollment 



5f f oik^CpmmuititY College (PubHc 2-Xear) Pell, enrollment ■ 

Heidelberg ,C:oli^e(iM?Ho.'f-yea^^ ' / Tuition and fees, enrollment 

SUNX^New-Paltz (PubHc Umversity) Pell, enrollment 

... • ^ , . , . . . ' 

, V. * ' - • 



FIGURE 6 

SCHOOLS WITH MORE THAN 
ONE- ERROR 



-23- 



ERIC 



26 



BEST copy AVAtUBLE 



Report- while a number of the' enrollment figures, must be summed off of the HEGIS' 
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points should be made^^con<;erning -the Pell and enrollment comparisons. 
Firsti, the Pell comparisons allowed a 1 percent tolerance before being declared in / .. 

error- This allowance was made because many schools' expenditures "do not match - ^ 
what the Pell Universe contaim for either current authorization or net expenditures.* 
This is due to the on^c^ing reconciliation process that continues for months after the . , 

• « 

award year. These adjjjstments are usually very minor and would account for the small 
discrepancies. ' . fi- 

k ■■ ■ . ' • . 

J 'V Secdnd, many schools seem to have filled in their total" enrollment figures for 

graduates and undergraduates directly off the HEGIS form. The total enrollment 

figur'tes from the'HEGiS include unclassified students. As a result, there really is no 

discrepancy that could be rectified by the institutional data collection. This problem 

is still an error. This error occurred in ^^6 schools. In addition, there were a ' * 

significant number of schools for whom the original HEGIS enrollment matches the 

amount on the FISAP, but for some reason, the HEGIS enrollment figure was changed. 

The reasons for changes in HEGIS have to do with editing processes that also -take 

place over several months. The impact of both of these discrepancies, uittlassif led ' ' 

students aiKl changes on HEGIS, is minor as a percentage of total enrollment. * ^ 

53 ANALYSIS PROBLEMS 

The results section indicates a number 9i problems that were encountered in the! 
analysis. The major ones are: 

f . • . . ' 

• Missing documents and data 

• Lack of specificity on audits and program reviews 

• . Unexplained changes in HEG^ figures »- . 

• Documentation of changes between the hard copy FISAP smd the FISAP 
data file. 

Th.e missing documents and data problems were discussed in the previous section. 

* ■ 

This problem is by far the most significant one restricting the analysis. Besides the " 
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missing FISAPs, there were 27 instances where tlWFfe was no HEGIS and 9 cases Where 
an amount for Pell could not be found. As mentioned earlier, the definition of a 
reporting unit is different across types of reporting. Therefore, even 'though a 
particular school filed a FISAP, there is no certainty that, using that particular 
school's identification number, it will show up on either the 'Pell Universe or the 
HEGIS. In addition, many proprietary schools do not file a HEGIS." The frequency of 
missing HEGIS forms include only those schools that provided undergraduate and * 
graduate enrollment on the FISAP, however, so there is no double-counting of missing 
forms. ■ , • ' 

^ "... * 

- ' The audits aiKi any program reviews pertaining to the 19S1-&;? year revealed very 
few schools with problems on their FISAPs. Only five of the 275 schools had program 
reviews relating to activities in that year. Of those, there was little or no specific"^ 
mention made of tfw application porUdfi of the FISAP. Any problems that did occur 
had to do with accounting for expenditures of Campus-Based funds. In the case of 
audits, 15 out of the*275 were not found. In part, this was due to the microfichiug 
process under way in the Audit Branch. -Of the 260 audits reviewed, 10 audits were 
found to have FISAP exceptions. T^e n^ajority of t^je audirexceptions had to do with 
the maintenance pf effort figures. In one school, the enrollment figures (531 rather 
than 538 undergraduates),; as well as the income grid of eligible aid. applicants were 
found to b« inaccurate. In two cases, the schools were under a general investigation 
fbr their administration of all student aid fuiKis. In all cases, ^hete was little or no 
specificity of what exactly was the probiefn. . 

In several cases there was a problem with changing values on the HEGIS form. 
Wh«i inspecting the HEGIS file, there vifere occurrences of HEGIS^ figures being 
manually changed from what was originally entered on the form by the institution. In 
most cases, the value on the FISAP matches the original value on the HEGIS. If there 
were ch^iges* made on the HEGIS, they -were not reflected on the FISA^. This 
situation brings to light an analytical issue: Does an err6r occur if the FISAP value . 
does not match the corrected HEGIS value? Since the editing of the HEGIS may be 
subsequent to the allocation of funds, is the institution in error? . Ttie reasons for 
changing HEGIS values are numerous but focus around cross-year edits, 'arithmetic 
errors, and so forth. The data collection for this analysis took into account arithmetic 
errors and entered only the correct value. However, th^re remained a number of 
unexplained changes of the HEGIS values. 
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The error reports that are returned to the institution for their review and 
correction do not always reflect all of the changes made to the FISAP file. There- 
were seven occurrences of discrepancies between the FISAP data file vMue ami the 
vai!;ie^^ the FISAP form. It i| unclear, at this point, however, whether the 
discrepancy arose out of the key entry error or from- a submitted change from the 
institution. In all cases, there were no error reports containing the dianged values. . 

6.0 CX)NCLUSIONS 

^ ... 

The results of the in-house data collectin showed that about 30 percent oi the 
schoorin our sample committed one ac more errors on the application portiwi of the 
FISAP. The majority of the errors are the enrollment figures that are supposed to 
come from the HEGI5 Fall Enroilmtait survey. Ccmtrary to a priori assumptions, the 
schools who committed error^ are not concentrated In the proprietary sector^ and 
schools do not tend to make many mistakes if they make one on the applicatiwu 
Insteaul, a disproportionate number of public scIkhsIs make errors althoughf^all typ<» of 
schools made errors. Also, very few schools vnade more than one error on the 
•'application, and of those who do, they make IK) more than two errors. 

Per'l'iaps more important than the numeric results are the discoveries made about 
the da-^a sources used. Sections 4 and 5 have documented the prd>lems eikountered 
when doing the data collection and analysis. In particular, the following problems 
were found: , • 

. • Missing documents — FlSAPs, audits, ^d program reviews 

• Difficulty of identifying institutions — no common idwitif ier other than 
nai%e, differences in reporting units 

' • Accuracy of the data — aUl data seem to be continually updated resulting in 
possible differences in different versions of the, same data file. 

Ail three of these problems limit the accuracy and usefulness of thia^ analysis because 
■'thefe may not exist a ♦*best value** for a particular data element on the application. In 
a larger sense, the existence of these problems may be an Indication th|^ any attempt 
to validate application data with other data must start one step b^k. Instead, of 

attacking the problem of how to validate the application, the other data must first be 

* ' ■ .. . ' ■ 
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cleaned up and then linked in some way to the Campus-Based System with a common 
identifier. Only then can one begin to validate data. 

i 

In terms of validating the data elements j)n the application, the following steps 
should be taken: 

• Rcstr^ the current edits ih the system to intemai consistency and cross- 
year FISAP data- " The current edits are good, but they dp not go far 
enough. Since the last year*s FISAP is "clean" it should be used to evaluate 
the values on the 'current yearns FISAP. This should be the first step of 
using other data files in the editing and updating process. 

• Begin to expand the use of other data lUes in h*os$<hecking when 

' .determiiwd to tfefe "clean." The PeU Universe file is a current JUe that is 
very accurate. It coul^ be vary useful for cross-checking. In the absence 
.of a very "clean" file, tolerances could be established within the system to 
allow minor discrepancies to pass. In this analysis, ^«n4 percent tolerance 
was allowed for Pell. About 3 percent of the institutions still had errors 
that exceeded this tolerance. If an exact match had been used, about 10 
percent of the institutions wovUd have been flagged. . . 

\ . . V ■ 

• Work with the ctevfelopers of other d^ta files to determine the validity of 
their data for purposes of validation. The editing cycles of a data file such 
as HEGIS may coincide with that of FISAP. If so, then an alternative needs 
to be found.vIt could be a dross-year HEGIS or something else. 

• Be creative with the kind of edits that are used. In this analysis, the 
combinatiwi of tuitioa and fees and enrollment was used as an error 

J measure. Others could be developed to check and cross-check the data in' 

- many different ways. v > 



♦ The number of errors in the application data requires that -som^f^ttepsfor. 
vaadating the data be planned. This an^ysis is a first step in 'that piannin|^ process 
because it has identified the kinds of problems one would meet in trying tb%ise data 
for validation purposes. . . ' 
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As part of the Campus-Based Quality Control Study, Advanced Technology wa^ asked 
by the Department of Education to comparje the values on the application {K>xtion of 
the FISAP to alternative sources of information. We found what appear- to be 
discrepancies in. the following data Items. Could you explain the. reason for the 
apparent discrepancy and provide us with any supportlnajjatumentation. ? 

Data Item FISAP ^ ' Other Source 

Tuition and fees • . ; - 

Pell Expenditures 



Income grid 

— UG Dependent 

- Grad. Dependent 

- UG Independent- 

* Grad. ]jndependent 



UG Enrollment 



Grad. Enrollment }(Z^. ^ - i^t^-^ 



\ 



Explanation H ( . ) - • pi ^PrP dedht^ u^^S 4pi:g./% ^-Q^ofv^ 
Explanation n ( ) - VprU^'S't ' i^Zi^ ^^U. PX . //^ 



Exoianation it^ ( ) - 
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— 
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SUPPLEMENTAL FISAP QUESTIONS FOR 



As part of the Campus-Based Quality Control Study, Advanced Technology was asked 
by the Department of Education to compare the values on the application portion of 
the FISAP to alternative sources of information, We iound what appear" to be 
discrepancies in the following data items. Could you explain the.reasor^^or the 
apparent discrepancy and provide us with any supporting documentation. 



Data Item 



Tuition and fees 



FISAP 



Other Source 



Pell Expenditures 



Income grid 



UG Dependent 
Grad. Dependent 
UG Independent 
Crad. Independent 



UG Enrollment 



Grad- Eniwllment 



* . 



J- •• 



Explanation iPl ( .) - - •'Tr^ (WrYvrl^gNA S.u. io rv\fti<'d[ 4a> V^6^S- 



tfV\irfiWtO a^ 'gueAffvo >Qf^0\U^gA 4^ aaM da^ ■iWNf&\\Yv\evv\'. Only -^vC^i 



Explanation ^2 ( 
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echnology' wai^asked 

cy xne ueparxment oi iioucation to compaFC the values on the application somon of • 
the FISAP to alternative sources disinformation. We found what appear- tc^Tte' 
discrepancies in the followingf data items. Could you , explain thcr reason for tij^e 
apparent discrepancy and provide us with any supporting documentation* 



■c 



V 



Data Item 



Tuition and fees 
Pell Expendituees 



FISAP 



Other Source 



■/ 



^come grid 

- UG Dependent 

- Grad. Depertdent 

- UG Independent 

- Grad. Inde|>endent 



UG Enrollment 
Grad.* Enrollment 
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SUPPLEMENTAL FISAP QUESTIONS FOR 



As part of the Campus-Baised Quality Control Study, Advanced Technology was asked 
by the Department of Education to compare the values on the application portion of 
^ th^^iFISAP to alternative sources of inlormatibm We found what appear" to be 
d^screpancieis in the following data items. Could you explain .the . reason for the 
apparent discrepahcy and provide us with any supporting documentation. 



Data Item 



Tuition and fees 



Pell Expenditures 



Income grid 

- UG Dependent 

- Grad. Dependent 

- UG Independent 

- Grad. Independent 



FISAP 



er Source 



UG Enrollment 
Grad. Enrollment 
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supplemental'fisap questions for 



As part of the Campus-Based Quality Control Study, Advanced Technology was asked 
by the Department of Education to compare the values on the application portion of 
the nSAP .to alternative sources of information. We found, what appear to be 
discrepancies in the 'following data items. Could ^you explain the reason for the 
apparent discrepancy and provide us with any supporting documentation* ^ ^ 

Data Item FISAP Other Source 

Tuition and fees 

Pell Expenditures 



r 



Income grid 



UG Dependent 

- Grad. Dependent ' 

- UG Independent - ^ 

- Grad. Independent . 

UG Enrollment ^ i^t ■ ^^rf 

Grad. Enrollment 
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SCHOOL 



George Wallace Community College 
Canada College 

Chaffey College ' 
Merritt College 

Skyline College ^ 
University of San Diego 
Colorado State University 
Southern Connecticut State College v 
Georgetown-Ur\iversity 
Florida State Univer9ity 
Indian River Comn>unity College 
Miami-Dade Community College 
Palm Beach Community College 
' Albany Junior College 
Albany^tate College 
Mercer University 
Ball State University 
Indiana State University 
Valparaiso University " 
St. Mary's College 
Delta School of Business 
Louisiana College 
Louisiana State University A&M 
Nicholis State College 
Bentiey College 
Boston University, 
Emmanuel College 
Fitchburg State College 
SUNY AdcT ^ ' ^ 

5UNY - Stony Brook 
Kent State University 
Draughon't^cfiool of Business 
Rogue Community College 
GanDon University 

m 

Pennsylvania' State University 



Ntew England Conservatory of Music 
N|ewberry Junior College 
Wheelock College 
Coppin State College 
-St. Joseph's College ^ 
Michigan State University 
Inver Hills Community Coltege 
Moorhead State College ' 
Soikhwest State College 
Un^ersity. of Minnesota - Duluth 
^University of Minnesota - Morris 
Missouri Western State College 
Delta State College 
Mississippi College 
Campbell University 
Qeveland Technical Institute 
North Carolina Central University 
Sacred Heart College 
Frankli%Pierce College ^ 
Atlantic Comrpumty College 
Ramapo College of New Jersey 
Rutgers, State University of New Jersey 
Trenton State College 
New Mexico State University 
Bard College 
Boricua College 
Dacmen College 
Suffolk Commimity College 
SUNY - New Paltz 
Heidelberg College 
Xavier University 
HUlsdaie Free Will College 
Bucks Community College 
Mercyhurst College 
Shippei^isburg University 
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SCHOOL 



iUniversity of Pennsylvania 
' Furman Univei|ity 
Rutledge Coiiege 
Tennessee State Univ^rsijty 
Texas State Technical Coiiege 
Bennington Coiiege : 
University of .Wisconsin - Milwaukee 



Anderson Coiiege 
Greenville Techfli<;aiCoilege 
University of S«0gh Cartlina 
Laredo Junior College ' 
Virginia Commonwealth University 
University of Washington 
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